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(54) Electronic imaging system using a removable software-enhanced storage device 



(57) A method for capturing and processing images 
in a digital camera (1) having a removable storage de- 
vice (24) for storing the images captured by the camera , 
comprising the steps of: 

(a) capturing and digitizing an image in response to 
a user input; 

(b) storing the captured image in an image buffer 
(1 8) capable of storing a plurality of images; 

(c) beginning the processing of the stored image us- 
ing a programmable processor (20); 



(d) if there is another user input during the process- 
ing step (c), interrupting the processing of the stored 
image in response to said user input, and repeating 
steps (a) and (b) for a second image; 

(e) continuing the processing of the first stored im- 
age after the completion of step (d); and 

(f) storing the processed image in the storage de- 
vice (24) and continuing with step (c) until ail images 
stored in the image buffer (18) have been proc- 
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Description 
Field of Invention 

[0001 ] This invention pertains to the field of electronic 
imaging and, in particular, to electronic imaging with an 
electronic still camera that utilizes a removable storage 
device for storing images. 

BACKGROUND OF THE INVENTION 

[0002] As understood in the prior art, a digital elec- 
tronic still camera is a device which uses an electronic 
sensor to capture an image; signal processing to modify 
the captured image signal, e.g. , as a function of ambient 
light, and to represent it numerically; and some storage 
device to preserve the numerical image data. Ordinarily, 
the image storage device is merely a receptacle for the 
image data. 

[0003] It is further known for a digital electronic still 
camera to use a removable storage device, such as an 
integrated circuit memory card, to store images. 
For instance, U.S. Patent No. 5,016,107 describes an 
electronic still camera utilizing image compression and 
providing digital storage in a removable memory card 
having a static random access memory. Software that 
controls the camera operation is located in non-remov- 
able programmable read only memory (PROM) integrat- 
ed circuits in the camera. In this camera, the integrated 
circuits in the removable memory card store image data 
and a directory locating the data. 
[0004] Other electronic camera systems described in 
the prior art record an image header along with the dig- 
ital image data. The header typically describes some 
characteristics about the image or the camera. For in- 
stance, in U.S. Patent No. 5,018,017, such header in- 
formation indicates the classification of the information; 
the imaging system in use; the date and time; the com- 
pression mode; as well as image-specific information, 
such as flash use, white balance data, exposure value, 
and shutter speed. Furthermore, U.S. Patent No. 
5,1 53,729 shows memory capacity, a free area pointer, 
a battery expiration date, and a write inhibit flag in the 
header area of a memory card. Typically, each system 
conveys such characteristic data from the electronic 
camera by way of the memory card (where the data is 
temporarily stored) to a peripheral device, such as a 
player. The object is to increase the downstream func- 
tionality of the system by allowing utilization of image- 
related data in subsequent processing of the image data 
in a downstream peripheral device. The camera or im- 
aging device generally utilizes little of the stored image 
characteristic data, one exception being data regarding 
the utilization of the storage device itself, i.e., how much 
memory space is remaining for further pictures. 
In the latter case, the camera accesses such data to dis- 
play utilization to the camera user. 
[0005] Coordination between a camera and a compu- 



ter is generally shown in US-A-5 ,475,441 , "Electronic 
Camera with Memory Card Interface to a Computer", 
which was filed on December 1 0, 1992 and assigned to 
common assignee with the present application. This ap- 
5 plication describes a camera that is specially shaped so 
as to plug directly into the memory card slot of a portable 
computer; the camera is then usable as a unit with the 
portable computer. The camera contains EPROM mem- 
ory with code for in-camera white balance and gamma 
10 correction, and also stores the compiled code which is 
used (by the computer) to operate the camera and to 
process the images from the sensor color filter array to 
obtain a full resolution, color corrected image. The im- 
age data is captured by the camera and directly down- 
's loaded to the computer, where it is processed with code 
obtained from the camera. As with prior memory card 
systems, the code is used downstream (in the compu- 
ter) to process image data. 

[0006] In a camera intended for acquisition of image 

20 data for use in a computer, such as described in US-A- 
5,475,441 , it is well understood that data corresponding 
to captured images may also be stored in a storage de- 
vice that is capable of being removed from the camera. 
Such a storage device is often considered analogous to 

25 film in a conventional film camera. Just as it is known to 
encode film with marks that are useful in downstream 
processing of the film, it is also known for an electronic 
image storage device to contain non-image data which 
is useful in modifying the image data. Following the film 

30 analogy, such additional data is useful in downstream 
processing, such as in a computer, to modify the image 
. data. Unlike film, however, the additional non-image da- 
ta may be read by a variety of peripheral devices, in- 
cluding the camera itself. This provides an opportunity 

35 to deal with a number of imaging problems. 

[0007] For instance, in most known electronic imaging 
systems, the software which controls the camera ordi- 
narily cannot be updated without purchasing an entirety 
new camera, containing a lens, CCD image sensor, etc. 

40 (Albeit, a few high end film cameras utilize a removable 
program module from which the camera can download 
exposure algorithms.) It is well known that computer 
programs are updated more regularly than computer 
hardware, since the software can be refined to provide 

45 higher performance (i.e. better image quality) or more 
features while using the same hardware. An electronic 
camera can use a programmable digital processor con- 
trolled by software. Such a camera offers an opportunity 
for software updates, if the system is designed to facil- 

so Rate such updates. What is needed is a method for soft- 
ware updates of the camera algorithms for higher per- 
formance, and for adding new features to a camera. 
Such a method should be quick and convenient for the 
user, preferably without requiring the use of additional 

55 storage modules. Existing memory devices are capable 
of limited two-way communication with an electronic 
camera, such as to report to the camera on memory 
space availability. An opportunity exists for enhancing 
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such "upstream" communication to provide altogether 
new applications for an electronic camera. 

SUMMARY OF THE INVENTION 

5 

[0008] The opportunity in the prior art for upstream uti- 
lization of the memory capability of a removable storage 
device is embodied, according to the invention in claim 
1, by an electronic imaging system including an elec- 
tronic camera for capturing and storing images in a re- 10 
movable storage device which is also preloaded with 
software for operating the imaging system. The imaging 
system includes an optical section for establishing the 
optical parameters of image capture, an image sensing 
section for electrically capturing an image provided by w 
the optical section, a signal processing section for 
processing the electrically captured image, a program- 
mable processor for controlling the sections of the sys- 
tem, and means for loading the programmable proces- 
sor with software preloaded in the removable storage 20 
device for affecting operation of one or more of the sec- 
tions of the system. 

[0009] Consequently, the invention provides an elec- 
tronic still photography system which can be controlled 
by software located on a removable memory (e.g., a 25 
memory card) used with an electronic still camera. 
Known cameras use the removable memory to store im- 
ages and data related to an individual image (date, color 
temperature of the illuminant, etc.) or data related to the 
camera (type of color filter array, sensor defect loca- 30 
tions, etc.) which is transferred to an image reproducing 
device. Except for data about memory space, the com- 
munication is one way. A camera according to this in- 
vention, however, uses the removable media to transfer 
software previously recorded on the memory card (such 35 
as improved camera firmware to process the image, or 
a special access code) to the camera. The advanta- 
geous effect is that the operation of the camera or play- 
back device can be improved some years after the cam- 
era is purchased, by using an "enhanced" memory card *o 
containing improved software for operating the camera 
or playback device. Moreover, such two-way communi- 
cation opens use of the storage device to other en- 
hancement possibilities, e.g., to modify image data in 
selected ways, to modify camera performance for spe- 45 
cial situations, or to communicate non-captured images 
(special overlays) to the camera. 

BRIEF DESCRIPTION OF THE DRAWINGS 

50 

[001 0] The prior art and the invention will be described 
in relation to the drawings, wherein 

Figure 1 is a diagram of a known electronic imaging 
system incorporating an electronic camera for in- ss 
putting images to a computer; 
Figure 2 is a block diagram of an electronic camera 
configured according to the invention to receive and 



process enhancement files; 
Figure 3 is a flow diagram of the steps involved in 
applying an enhancement algorithm to the camera 
of Figure 2; 

Figure 4 is a block diagram of the capture chain 
showing the points at which processing according 
to the invention may occur; 
Figure 5 is a flow diagram of the steps involved in 
processing an image sometime after storage in the 
camera of Figure 2; 

Figure 6 is a flow diagram of the steps involved in 
interrupt processing in the camera of Figure 2; 
Figure 7 is a flow diagram of the steps involved in 
controlling access to processing in the camera of 
Figure 2; 

Figure 8 is a pictorial illustration of the application 

of pre-existing overlay images to images captured 

by the camera of Figure 2; 

Figure 9 is a pictorial illustration of the use of an 

electronic viewfinder in the camera of Figure 2 to 

frame the captured image with respect to an overlay 

image; 

Figure 10 shows an additional embodiment of an 
electronic camera configured according to the in- 
vention; and 

Figure 11 shows an electronic imaging system in- 
corporating the camera of Figure 1 0 by means of a 
cable connection. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

[0011] Because electronic still cameras employing 
electronic sensors are well known, the present descrip- 
tion will be directed in particular to elements forming part 
of, or cooperating more directly with, apparatus in ac- 
cordance with the present invention. Elements not spe- 
cifically shown or described herein may be selected 
from those known in the art. 

[0012] Figure 1 shows a known electronic still photog- 
raphy system useful for inputting images to a computer. 
The system includes an electronic camera 1 with an 
electronic sensing section la, a digital processing sec- 
tion 1 b, a memory card slot 2, a removable memory card 
3, a computer 4 with a built-in memory card reader 5, 
and a printer 6. Images are captured by the sensing sec- 
tion la in the camera 1 , manipulated in the processing 
section lb, stored on the memory card 3, and trans- 
ferred from the card 3 to the computer 4. The software 
used to control the sensing and processing sections la 
and lb in the camera is entirely located in firmware mem- 
ory 7 in the camera 1 and cannot be updated without 
opening the camera cover, unsoldering the firmware 
memory circuit, and replacing it with a new circuit con- 
taining updated code. The application software used to 
control the playback operations of the computer is sup- 
plied, at least in part, by the camera manufacturer. It is 
stored in a computer hard drive 8, and downloaded to a 
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computer random access memory (RAM) memory 9a, 
where trie program is used by the central processing un it 
(CPU) 9b. To update this software, the user must be pro- 
vided with a new program on a floppy disc, which can 
be transferred to the hard drive 8. For example, the flop- 
py disk may contain a "plug-in" module for an imaging 
software application provided with the camera. The 
plug-in contains both code to allow the images to be 
downloaded from the camera to the computer, and soft- 
ware code to create color images from the digitized cam- 
era image data. The software code implements algo- 
rithms for processing the color filter array sampled im- 
age data supplied by the camera. These algorithms in- 
clude color interpolation, color balance, and color cor- 
rection. A typical color interpolation algorithm is de- 
scribed in greater detail in US-A-5, 373, 322, "Apparatus 
and method for adaptively interpolating a full color im- 
age utilizing chrominance gradients", filed June 30, 
1993, which is assigned to Eastman Kodak Company. 
[0013] The invention is based on the recognition that 
the storage device, i.e., the memory card 3 in Figure 1 , 
need not be primarily oriented to storage of data for 
downstream (i.e., computer) processing. The storage 
device may additionally contain files (data, code, etc.) 
which are capable of upstream modification of image 
data and camera performance. This provides an oppor- 
tunity for affecting image data during the capture and 
manipulation stages prior to storage. Furthermore, 
these files may be accessed automatically by the cam- 
era, or selected by the user by means of appropriate 
intervention through the camera. Therefore, the remov- 
able image storage device is capable of two-way com- 
munication with the camera. That is, the card slot inter- 
face is not limited to "dumping" image data downstream 
to the removable storage device, but also allows com- 
munication from the removable storage device up- 
stream to the camera as well. Inasmuch as the files are 
software which enhance the operation of the camera, 
the digital storage device may be viewed as a software- 
enhanced device, and the files are hereinafter referred 
to as enhancement data files. 
[0014] A number of advantages arise from such two- 
way communication. Enhancement files can be includ- 
ed in the digital storage device which are not in them- 
selves images. Image processing software, look-up ta- 
bles, matrices, compression tables, dynamic range op- 
timization tables, and other files capable of affecting the 
captured image data can be included in the digital stor- 
age device. Typical algorithms which can be modified, 
or updated, thus include color filter array interpolation 
algorithms, noise reduction algorithms, edge sharpen- 
ing algorithms, color reproduction algorithms, compres- 
sion algorithms, or dynamic range optimization algo- 
rithms. In this manner, the camera need only be capable 
of executing the functions and transformations con- 
tained in the storage device, and need not require the 
storage capacity to contain all information necessary to 
execute the processing of the image data. Such non- 



image files, which cause processing of the captured im- 
age data, may process the captured image data such 
that the most "true-to-scene" reproduction is achieved, 
or instead provide special effects of the kind which 
5 change the image into some desired alteration of the 
original scene. 

[0015] Another type of non-image file which can be 
contained in the digital storage device enhances the op- 
eration of the camera. Enhancement files can be includ- 
10 ed which control capture parameters such as exposure 
time, aperture setting, flash range output, and so on. A 
particular set of control parameters can be used to tailor 
the capture parameters to the type of scene or subject 
being captured. This type of file might also allow diag- 
is nostictestsof thecamerato insure proper operation and 
to detect fault conditions in the camera system. A further 
type of enhancement file contained in the storage device 
comprises pre-existing image data files, i.e., files with 
images not captured by the camera system. Such files 

20 are accompanied by code which instructs the camera 
on how to merge the pre-existing image files wfth those 
captured by the camera. In this manner graphic overlays 
and text can be superimposed on the captured images 
by the camera. Proper execution of such a feature might 

25 include a user interface for selection of pre-existing im- 
age and overlay combinations, and also a viewfinder 
which helps the user to compose the captured image so 
as to complement the selected overlay. 
Whetherfor image processing, enhanced performance, 

30 or pre-existing images, the algorithms themselves are 
conventional and not themselves part of this invention. 
However, where they are stored, and how they are used, 
is within the scope of the invention. In that connection, 
another function of the invention is to store replacement, 

35 or updated, software files in the digital storage device 
so that software updates can be added to the camera 
after its initial purchase by the subsequent purchase of 
appropriately-enhanced storage devices. 
[0016] The invention also concerns the sequence of 

40 execution of operations made possible by files con- 
tained in the image storage device. It is anticipated that 
the capture of images may occur in a rapid and/or ran- 
dom fashion. It may be undesirable to interrupt image 
processing to initiate another capture. If that is the case, 

45 processing above and beyond that which is necessary 
to achieve initial storage would not have to occur until 
the camera is no longer in a "ready" mode. That is, no 
processing takes place until the camera is no longer be- 
ing used to capture images and is placed in an "off or 

so "sleep" mode. Alternatively, the camera could interrupt 
processing operations to carry out capture operations. 
Once the capture is complete, the camera can return to 
processing operations. 

[0017] Figure 2 shows an electronic still camera ca- 
55 pable of using a "software enhanced" storage device as 
taught by the invention. The electronic still camera is 
divided generally into an input section A and a compres- 
sion and recording section B. The input section A in- 
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eludes an exposure section 10 for directing image light 
from a subject (not shown) toward an image sensor 12. 
A flash unit 11 may be used to illuminate the subject. 
Although not shown, the exposure section 10 includes 
conventional optics for directing the image light through 
a diaphragm, which regulates the optical aperture, and 
a shutter, which regulates exposure time. The sensor 
12, which includes a two-dimensional array of pho- 
tosites corresponding to picture elements of the image, 
is a conventional charge-coupled device (CCD) using, 
e.g., either well-known interline transfer or frame trans- 
fer techniques. The sensor 12 is exposed to image light 
so that analog image charge information is generated 
in respective photosites. The charge information is ap- 
plied to an output diode 14, which converts the charge 
information to analog image signals corresponding to 
respective picture elements. The analog image signals 
are applied to an A/D converter 16, which generates a 
digital image signal from the analog input signals for 
each picture element. 

[0018] The digitalsignalsareappliedtoan imagebuff- 
er 1 8 for storing one or more images, albeit shown in 
Figure 2 as a random access memory (RAM) with stor- 
age capacity for a plurality of still images. A programmed 
control processor 20 generally controls the in put section 
A of the camera by initiating and controlling exposure 
(by operation of the diaphragm and shutter (not shown) 
in the exposure section 10), by controlling the flash unit 
11, by generating the horizontal and vertical clocks 
needed for driving the sensor 1 2 and for clocking image 
information therefrom, and by enabling the A/D convert- 
er 16 in conjunction with the image buffer 18 for each 
signal segment relating to a picture element. (The con- 
trol processor 20 would ordinarily include a microproc- 
essor coupled with a system timing circuit.) Exposure 
control would typically be effected by using ambient light 
information from a photocell 19. 
[001 9] Once a certain number of digital image signals 
have been accumulated in the image buffer 18, the 
stored signals are applied to a programmed digital sig- 
nal processor 22, which controls the throughput 
processing rate for the compression and recording sec- 
tion B of the camera. The algorithms and other operating 
code used by the processor 22 are stored in the algo- 
rithm memory 28. The digital signal processor 22 com- 
presses each still image stored in the image buffer 18 
according to a known image compression algorithm, 
such as the well-known JPEG (Joint Photographic Ex- 
perts Group) discrete cosine transformation-based 
compression algorithm. The processor 22 applies a 
compression algorithm from the memory 28 to the digital 
image signals, and sends the compressed signals to a 
removable storage device via an interface 26. While a 
memory card 24 is shown in this system as the storage 
device, other devices may be used, such as a floppy 
disk magnetic medium, a small hard drive, or optical 
storage (in the latter cases, suitable conventional read- 
ing/writing apparatus would be provided in the camera, 



e.g., magnetic or optical read/write head, etc.) A repre- 
sentative memory card is a card adapted to the PCMCIA 
card interface standard, such as described in the PC 
Card Standard, Release 2.0, published by The Personal 
5 Computer Memory Card International Association, Sun- 
nyvale, California, September 1991. Moreover, while 
compression is shown in the camera of Figure 2, com- 
pression is not a necessary feature of the invention, and 
uncompressed data could be stored instead. 
10 [0020] The memory card 24 contains solid state mem- 
ory 24a, such as Flash EPROM memory, which the card 
uses to store image data files. In addition, the memory 
card 24 contains additional memory to store enhance- 
ment data files 24b to modify the camera operation, or 

15 the images captured by the camera, or to provide pre- 
existing overlay images. These software enhancements 
are normally programmed by the card manufacturer be- 
fore the card is sold to the user, and can be stored in the 
Card Information Structure (CIS) of the PCMCIA Format 

20 (as described in the aforementioned Release 2.0). The 
"software enhancement" code may be stored in sepa- 
rate non-erasable memory on the card 24, or in a portion 
of the memory space of the memory 24a used to store 
the image data. 

25 [0021] The input section A operates at a rate com- 
mensurate with normal operation of the camera while 
compression, which may consume more time, can be 
relatively divorced from the input rate. The exposure 
section 10 exposes the sensor 12 to image light for a 

30 time period dependent upon exposure requirements, for 
example, a time period between 1/1000 second and 
several seconds. The image charge is then swept from 
the photosites in the sensor 1 2, converted to a digital 
format, and written into the image buffer 1 8. The repe- 

35 tition rate of the driving signals provided by the control 
processor 20 to the sensor 1 2, the A/D converter 1 6 and 
the buffer 1 8 are accordingly generated to achieve such 
a transfer. The processing throughput rate of the com- 
pression and recording section B may be determined by 

40 the character of an image, i.e., the amount of detail ver- 
sus redundant information, and the speed of the digital 
signal processor 22. 

[0022] An operation display panel 30 is connected to 
the control processor 20 for displaying information use- 

45 ful in operation of the camera. Such information might 
include typical photographic data, such as shutter 
speed, aperture, exposure bias, color balance (auto, 
tungsten, fluorescent, daylight), high/low resolution, low 
battery, low light, exposure modes (aperture preferred, 

so shutter preferred), and so on. Moreover, other informa- 
tion unique to this type of camera is displayed. For in- 
stance, the memory card 24 would ordinarily include a 
directory signifying the beginning and ending of each 
stored image. This would show on the display 30 as ei- 

55 ther (or both) the number of images stored or the 
number of image spaces remaining, or estimated to be 
remaining. In certain applications, an electronic view- 
finder 29 would be included for displaying images either 
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before or after storage. 

[0023] The control processor 20 also generates im- 
portant information about the condition and capability of 
the memory card 24. Specifically, the interface 26 is que- 
ried for the presence of a card 24 and, if no card is con- 
nected, a "no card" display is produced on the operation 
display 30. Likewise, if a card is present but it is full of 
images, a "card full" display is produced. While not 
shown, the display 30 may show the number of "remain- 
ing images." If enhancement files 24b are present on 
the card 24, they can be identified on the operation dis- 
play 30. For instance, performance enhancements, 
such as special set-up for portraiture, can be identified; 
processing enhancements, such as special effects, can 
be identified; and overlay enhancements, such as birth- 
day or holiday picture borders, can be identified. User 
intervention in order to specify the several enhancement 
opportunities, or to set conventional capture parame- 
ters, is obtained from a user inputs section 21 . Such in- 
puts could be one or more selector switches operable 
alone or in conjunction with prompts from the operation 
display 30 or the electronic viewfinder 29. 
[0024] The camera also contains firmware memory 
32 with operating code for the camera, as well as RAM 
instruction memory 31 connected to receive code from 
the memory card 24. When the memory card 24 is in- 
serted into the camera, the processor 20 uses the 
firmware algorithms in the memory 32 to determine if 
the memory card 24 contains "software enhancements" 
which can be used by the camera. If not, the firmware 
algorithms supplied with the camera in the memory 32 
are used by the programmable processor 20. If the card 
does contain appropriate "software enhancements", 
however, these are downloaded from the enhancement 
file section 24b of the card 24 to the RAM instruction 
memory 31 and used by the programmable processor 
20 in place of some portion of the firmware algorithms 
supplied with the camera from the firmware memory 32. 
In this way the camera can execute new, improved al- 
gorithms which provide higher quality pictures or special 
features. More than one algorithm file may also be 
downloaded, or different functions may be combined in- 
to a single file. Any combination of image processing 
files may be used. 

[0025] After the images from the camera are stored 
on the memory card 24, the card is inserted into the com- 
puter memory card reader 5 (see Rgure 1). In certain 
applications where the memory card may contain "soft- 
ware enhancements" which can be used by the compu- 
ter, the computer uses the software supplied with the 
camera to check if the memory card contains such al- 
gorithms. If not, the software algorithms supplied with 
the camera (and stored in the drive 8) are downloaded 
to the computer RAM memory 9a used by the CPU 9b 
to process the image. If the card does contain appropri- 
ate "software enhancements", however, these are 
downloaded from the card to the RAM instruction mem- 
ory 9a and used by the CPU 9b in place of some portion 



of the software algorithms from the hard drive 8. This 
allows a quick and convenient way of updating the al- 
gorithms used by the computer, without requiring a spe- 
cial update disk to be distributed to users. These image 
5 processing algorithms can be applied by the camera ei- 
ther during initial processing, when the image is first 
captured, processed, and stored, or post-capture. 
[0026] According to the invention shown in Figure 2, 
the images are transf erred from the camera to the com- 

10 puter via the removable memory, rather than a cable in- 
terface. Therefore, the camera does not have to be in 
the same location as the computer. This makes it pos- 
sible to view the images from the camera in many dif- 
ferent locations, on many different computers, as long 

is as the computer has the ability to both accept the re- 
movable memory and correctly process the image data. 
Since the removable memory can be used to store code 
for processing the image data in the computer, this code 
can include the color interpolation, color balance, and 

20 correction algorithms described in connection with Fig- 
ure 1 , or future improved versions of these algorithms. 
The advantage of storing the algorithms on the remov- 
able memory is that it eliminates the need for a separate 
means of supplying the code, such as the floppy disk. 

25 Supplying the algorithms along with the images on the 
removable memory, such as a PCMCIA card, makes it 
possible for any computer capable of reading the image 
data from the card to also download and utilize the al- 
gorithms needed to process the image data in order to 

30 create a full color image. This makes it convenient to 
view the images on almost any computer which includes 
a PCMCIA card connector, without requiring that the 
computer be supplied with a special plug-in to provide 
the image processing code, such as for a color filter ar- 

35 ray Interpolation. 

[0027] Apart from updating camera algorithms, the 
enhanced algorithm files might be tailored in a specific 
manner for memory cards sold for particular applica- 
tions. In electronic still imaging systems, the resolution 

40 and/or photographic speed of the camera may be al- 
tered by, e.g., subsampling and/or electrical gain adjust- 
ments. A type of non-image data file providing such ad- 
justments would be included in image data storage de- 
vices which are designed for specific capture applica- 

45 tions. In the same manner that certain film speeds are 
selected for films, an image storage device might be se- 
lected for action photography, portraiture, or infant pho- 
tography. An appropriate set of parameters for image 
capture would be incorporated in the associated non- 
50 image data file. One possible application is portraits. In 
this application, the non-image data file would provide 
that the color reproduction should be optimized for skin 
tones, the edge sharpening should not be too severe, 
and the color filter array algorithm should minimize color 

55 aliasing at the expense of reduced sharpness. Other ap- 
plications could also be envisioned. 
[0028] The steps involved in applying an image 
processing algorithm are shown in the flow diagram of 
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Figure 3. When the camera is ready to capture images, 
the processor 20 within the camera checks if there are 
software enhancements in the enhancement file 24b in 
the card 24. If such files are present, the processor 20 
downloads the processing algorithm(s) in the enhance- 
ment file 24b contained in the removable memory card 
24. If there is more than one algorithm, the user would 
preselect the appropriate one from the user input sec- 
tion 21 , or the camera would select one according to the 
detected picture taking situation; absent any selection, 
the camera could use a default (or no) algorithm. Once 
the exposure is made, processing may commence. Fig- 
ure 4 is a block diagram of the capture chain showing 
several points at which processing may occur. The al- 
gorithms may be applied at point 40 after the image data 
is digitized. Processing according to the downloaded al- 
gorithm may alternatively take place as the image data 
is originally being processed for initial storage at points 
42 or 44, or may be applied at a later time after the initial 
storage of the image is complete at point 46. Figure 5 is 
a flow diagram showing the steps of subsequent 
processing at point 46 (Figure 4), including after a time 
lapse. There may be utility in storing the initial capture 
in raw form so that different algorithms can be applied 
to achieve the most desirable result. The processing al- 
gorithms themselves, examples of which were de- 
scribed earlier, are conventional and not themselves 
part of this invention. 

[0029] For all image data manipulations, it may be ad- 
vantageous to postpone image data processing until 
some time after the capture is made as generally implied 
by processing point 46 in Figure 4. This is to say that 
any processing which is done to the image beyond that 
which is necessary for initial storage may occur when 
the camera is in the "sleep" or "off" mode. This may be 
desirable so that the camera can be ready for the next 
exposure in the minimum time possible, and not be bur- 
dened with post-processing of the image during a time 
when image capture may still occur. 
[0030] An alternative to delayed processing is to com- 
mence processing as soon as the capture is complete, 
and to interrupt the processing when the user indicates 
another capture must take place, as shown in Figure 6. 
This is an acceptable means of processing the images, 
but may become complex if a number of images are cap- 
tured in rapid succession. Delayed processing, as 
shown in Figure 5, may be preferred. A flag in the image 
file header may be set to indicate if the image has been 
processed or not. A camera in the midst of a processing 
operation may be programmed to return to "ready" 
mode only after all processing for the current image is 
complete. 

[0031] In addition to storing complete software algo- 
rithms, the software enhancement located in the en- 
hancement file 24b on the card 24 can be codes which 
are used to allow access to certain features of the soft- 
ware already located in the camera (or computer), or to 
select among specific options for the software located 



in the camera (or computer). An example of the use of 
such access codes is shown in the flowchart of Figure 
7. When the card 24 is inserted into the camera 1 , the 
camera checks to determine if the card is a normal mem- 
5 ory card, or a "software enhanced" card. For PCMCIA 
cards, the use of the "software enhanced" card might be 
indicated, for example, by the use of a digital encryption 
key as part of the Card Information Structure. The card 
also includes special access codes for exposure index, 

10 color/monochrome, color reproduction and portrait. The 
camera may include variable gain corresponding, e.g., 
to ISO exposure indexes of 50, 100, 200, and 400. For 
normal cards, the camera speed is set at ISO 100. For 
software enhanced cards, the camera checks to see 

15 which speed should be used for the photos, depending 
on which version of the card the user has inserted into 
the camera. In this embodiment, the camera includes 
algorithms in the firmware memory 32 necessary for var- 
iable gain, and for storage of either color images or the 

20 monochrome record from the color image sensor 12. 
For software enhanced cards, a "black and white" bit of 
the card is checked, to see which algorithm should be 
used. The user then takes the desired photos, and in- 
serts the card into the computer to print the images in 

25 the printer 6. When the card 24 is inserted into the com- 
puter 4, the computer checks the Card Information 
Structure to determine if the card is a normal memory 
card, or a "software enhanced" card. For normal cards, 
the normal processing is used. For "software enhanced" 

30 cards, more elaborate color reproduction algorithms 
might be used, to provide higher quality images. The 
card's "portrait bit" is also checked, to see if the user has 
selected a card for taking portraits. If so, the special 
processing is used to soften the image and reduce noise 

as which might appear as facial blemishes. The image is 
then printed by the printer. 

[0032] "Pre-exposed" image data files can also be 
resident in the enhancement file 24b in the removable 
image data storage device. These files would include 

40 image templates or overlays for combination with user- 
captured images, as shown pictorially in Figure 8. These 
files would likely be computer generated surrounds hav- 
ing the purpose of enhancing the images captured by 
the user for particular situations. The manner in which 

45 these files are combined with user captured images can 
be automatic or with user intervention through the user 
input section 21 . In the camera, the processor 20 would 
retrieve the overlay and the digital processor 22 would 
insert the user captured image into the overlay sur- 

50 round. A given removable image storage device might 
be labeled as a Christmas Album, for example, and the 
camera would insert the user captured images into the 
seasonal templates without input from the user. The 
camera may prompt the user to frame the image appro- 

55 priateiy with cues in the electronic viewfinder 29, as 
shown in Figure 9. An outline of the template might ap- 
pear in the viewfinder 29 to assist with framing. The 
camera should be capable of understanding where the 
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"pre-exposed" overlay will appear In the final combined 
image in order to assist the user with framing. A file con- 
veying this information may accompany the overlay data 
file, the information may be contained in the overlay data 
file header, or the camera may be capable of interpreting 
the location of the overlay by reading the overlay data 
file itself. Many different types of overlays may be used. 
The user could select removable image storage devices 
expressly for different seasonal or special events such 
as birthdays or sporting championships. 
[0033] Implementation of graphic overlays can also 
be accomplished with user input. The user would select 
images and overlays according to personal preference 
after observing them in the viewfinder 29 or on a views- 
creen (not shown) included in the camera. This ability 
implies an ability to browse both images and overlays 
contained on the image storage device by means of the 
camera. The camera must be capable of reading and 
displaying images from the removable storage device, 
and allowing the user to select both independently. In 
certain applications, more than one captured image 
could be inserted into an overlay. Alternatively, while a 
single image may be inserted, the framing of the image 
in the viewfinder relative to the overlay need not be in 
one-to-one correspondence between the captured im- 
age and the displayed image. That is, the captured im- 
age may be reduced so that the whole captured image 
is visible in the display through the overlay. Likewise, 
the captured image could be rotated or otherwise trans- 
formed as to position before being inserted into the over-, 
lay. Moreover, while the captured image may be shown 
through the overlay in the viewfinder, the camera may 
not actually combine these images, but rather create a 
script file which would direct the computer to do the 
proper combination. With adequate internal memory in 
the buffer 18 or multiple storage inputs (e.g., card slots 
26), the camera might also allow selection of an overlay 
from one removable storage device to be combined with 
a captured image resident on another. 
[0034] While the invention has been described with 
particular reference to several embodiments, it will be 
understood by those skilled in the art that various chang- 
es may be made and equivalents may be substituted for 
elements of the embodiments without departing from in- 
vention. For example, instead of using a removable 
memory card 24, the image data files 24a and enhance- 
ment files 24b could be stored in non-removable elec- 
trically programmable non-volatile memory located in- 
side the camera. As shown in Figure 11, image data 
would be downloaded from the non-volatile image mem- 
ory 35 in the camera 1 to the computer 4 over a standard 
electrical computer interface cable 38, such as a RS- 
232 or SCSI interface connection. The enhancement file 
would be supplied on a floppy disk, downloaded to the 
computer 4, and then uploaded by the computer 4 to an 
electrically erasable firmware memory 36 in the camera 
1 via the same interface cable 38. This additional em- 
bodiment of the invention is further shown in Figure 10 



in relation to an interface section 34 through which im- 
ages are transferred to the computer 4 and operating 
software is uploaded from the computer to the electri- 
cally erasable firmware memory 36 in the camera. Fig- 

s ure 11 generally shows the connection of the camera of 
Figure 1 0 to a computer 4 by means of a standard elec- 
trical computer interface cable 38. In addition, many 
modifications may be made to adapt a particular situa- 
tion without departing from the essential teachings of the 

w present invention. It is accordingly intended that the 
claims shall cover all such modifications and applica- 
tions as do not depart from the scope of the invention 
as claimed. 

15 

Claims 

1 . A method for capturing and processing images in a 
digital camera (1) having a removable storage de- 

20 vice (24) for storing the images captured by the 
camera , comprising the steps of: 

(a) capturing and digitizing an image in re- 
sponse to a user input; 
25 (b) storing the captured image in an image buff- 

er (18) capable of storing a plurality of images; 

(c) beginning the processing of the stored im- 
age using a programmable processor (20); 

(d) if there is another user input during the 
30 processing step (c), interrupting the processing 

of the stored image in response to said user in- 
put, and repeating steps (a) and (b) for a sec- 
ond image; 

(e) continuing the processing of the first stored 
35 image after the completion of step (d); and 

(f) storing the processed image in the storage 
device (24) and continuing with step (c) until all 
images stored in the image buffer (18) have 
been processed. 

40 

2. The method of Claim 1 wherein the interrupting step 
(d) further comprises the step of: 

storing a partially processed image such that 
continued processing in step (c) is resumed where 
45 the interuption occurred. 
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